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(54) Display apparatus for vehicle 

(57) In a display apparatus for a vehicle, when an 
auto mode is selected, an obstacle closest to the vehi- 
cle is determined on the basis of data that represents an 
output signal from an obstacle sensor, the determined 
obstacle and a radiation heat source (pixel group) 



included in an image sensed by an infrared light camera 
are associated with each other, and only an image cor- 
responding to the obstacle closest to the vehicle is dis- 
played on a display. 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to a display 
apparatus for a vehicle and, for example, to a display 
apparatus which is suitably mounted on an automobile 
as a representative vehicle. 

BACKGROUND OF THE INVENTION 

[0002] Conventionally, a technique for displaying an 
image ahead of a vehicle, which is sensed using infra- 
red rays, on a display provided in front of a driver's seat 
so as to assist driving of a driver in a maneuvering envi- 
ronment with poor visibility such as nighttime, heavy 
fog, or the like has been proposed in, e.g., Japanese 
Patent Laid-Open Nos. 60-231193, 6-247184, 10- 
230805, or the like. 

[0003] The display apparatuses proposed by these 
prior arts can effectively assist driver's vision. 
[0004] However, when the sensed infrared ray 
image includes a plurality of obstacles, it is difficult for 
the driver to recognize the distances between the indi- 
vidual obstacles and his or her vehicle. For this reason, 
it is hard to determine an obstacle which the driver must 
watch out most, and such display may often have 
adverse influences on the driving of the driver. 
[0005] When a far obstacle is sensed, its image is 
displayed to have a small size in the infrared ray image. 
For this reason, the driver cannot recognize the obsta- 
cle displayed on the display screen at a glance, and 
such image cannot effectively assist driver's vision. 
[0006] When the sensed infrared ray image is 
always displayed irrespective of the maneuvering envi- 
ronment, such display may have adverse influences on 
the driving of the driver. 

SUMMARY OF THE INVENTION 

[0007] The present invention has been made in 
consideration of the aforementioned conventional prob- 
lems, and has as its object to provide a display appara- 
tus for a vehicle, which can display an obstacle which 
the driver must watch out, so that the driver can easily 
recognize the displayed image. 
[0008] In order to achieve the above object, a dis- 
play apparatus for a vehicle according to the present 
invention is characterized by the following arrangement. 
[0009] That is, a display apparatus for a vehicle of a 
present invention is a display apparatus for a vehicle, 
which is provided to a vehicle, and comprises an image 
sensing device (3, 23, 43) for sensing an image of a 
landscape ahead of the vehicle using infrared rays, a 
display (8, 28, 48) for displaying the image sensed by 
the image sensing device at a position in front of a 
driver's seat of the vehicle, and distance detection 
means (2, 22, 42) for detecting an obstacle present 



ahead of the vehicle, and detecting a distance between 
the obstacle and the vehicle, characterized by compris- 
ing display control means (1, 21, 41) for changing a dis- 
play mode of the display in accordance with a detection 

5 result of the distance detection means. 

[0010] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) computes distances 
to obstacles contained in the image sensed by the 

w image sensing device (3) on the basis of that image and 
the detection result from the distance detection means 
(2), executes a predetermined image processing of the 
image so as to display only the obstacle with the short- 
est computed distance to the vehicle, and displays an 

15 image that has undergone the image processing on the 
display (8). 

[0011] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) computes distances 

20 to obstacles contained in the image sensed by the 
image sensing device (3) on the basis of that image and 
the detection result from the distance detection means 
(2), executes a predetermined image processing of the 
image so as to emphasize and display the obstacle with 

25 the shortest computed distance to the vehicle com- 
pared to other obstacles, and displays an image that 
has undergone the image processing on the display. 
[0012] Furthermore, in the apparatus arrangement, 
the display control means preferably executes the fol- 

30 lowing processing as the predetermined processing: 

a processing for changing contrast between a par- 
tial image that represents the obstacle closest to 
the vehicle, and a surrounding image thereof; 

35 • a processing for setting a display color of a partial 
image that represents the obstacle closest to the 
vehicle to be different from a display color of a par- 
tial image that represents another obstacle; 
a processing for setting a luminance of a partial 

40 image that represents the obstacle closest to the 
vehicle to be higher than a luminance of a partial 
image that represents another obstacle; or 
a processing for emphasizing and displaying an 
outline of a partial image that represents the obsta- 

45 cle closest to the vehicle. 

[0013] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) computes distances 
so to obstacles contained in the image sensed by the 
image sensing device (3) on the basis of that image and 
the detection result from the distance detection means 
(2), extracts a partial image that represents the obstacle 
with the shortest distance to the vehicle, executes a pre- 
ss determined image processing for enlarging the 
extracted image, and displays an image that has under- 
gone the image processing on the display (8). 
[0014] In a preferred embodiment of the display 
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apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) extracts a partial 
image that represents a person from an overall image 
sensed by the image sensing device (3), computes a 
distance between the vehicle and the person corre- 5 
sponding to the partial image on the basis of the overall 
image and the detection result from the distance detec- 
tion means (2), executes a predetermined image 
processing of the overall image so as to emphasize and 
display the partial image that represents the person with to 
the shortest computed distance to the vehicle com- 
pared to partial images representing other obstacles, 
and displays an image that has undergone the image 
processing on the display (8). 

[0015] In a preferred embodiment of this apparatus 15 
arrangement, preferably, the display control means 
determines an obstacle, a radiation temperature of 
which falls within a predetermined temperature range, 
to be a person on the basis of radiation temperatures of 
obstacles included in the overall image, prior to extrac- 20 
tion of the partial image that represents the person from 
the overall image. 

[0016] More preferably, the apparatus arrangement 
further comprises maneuvering environment detection 
means (4, 5) for detecting a maneuvering environment 25 
of the vehicle, and the display control means corrects 
the predetermined temperature range in accordance 
with the maneuvering environment detected by the 
maneuvering environment detection means. More spe- 
cifically, the following processing is preferably done: 30 

the maneuvering environment detection means (5) 
detects as the maneuvering environment an atmos- 
pheric temperature of the vehicle, and the display 
control means shifts the predetermined tempera- 35 
ture range toward higher temperature with increas- 
ing atmospheric temperature detected by the 
maneuvering environment detection means; or 
the maneuvering environment detection means (5) 
detects as the maneuvering environment calendar 40 
information or time information, and the display 
control means corrects the predetermined temper- 
ature range on the basis of the calendar information 
or time information detected by the maneuvering 
environment detection means. 45 

[0017] Alternatively, as the emphasized display 
mode in the display apparatus for a vehicle of the 
present invention, the display control means preferably 
makes the following display as the predetermined 50 
image processing: 

to change contrast between a partial image that 
represents the person closest to the vehicle, and a 
surrounding image thereof; 55 
to set a display color of a partial image that repre- 
sents the person closest to the vehicle to be differ- 
ent from a display color of a partial image that 



represents another obstacle; 
to set a luminance of a partial image that repre- 
sents the person closest to the vehicle to be higher 
than a luminance of a partial image that represents 
another obstacle as the predetermined image 
processing; 

to emphasize and display an outline of a partial 
image that represents the person closest to the 
vehicle; or 

to select a predetermined number of distances 
which are computed between the vehicle and per- 
sons corresponding to the partial images in 
ascending order of distance to the vehicle, and 
emphasize and display the partial images that rep- 
resent the persons corresponding to the selected 
distances compared to the partial images that rep- 
resent the other obstacles, as the predetermined 
image processing. 

[0018] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) extracts partial 
images that represent persons from an overall image 
sensed by the image sensing device (3), computes dis- 
tances between the vehicle and the persons corre- 
sponding to the partial images on the basis of the 
overall image and the detection result from the distance 
detection means (2), and displays the partial image that 
represents the person with the shortest computed dis- 
tance to the vehicle on the display, and also the com- 
puted distance to the vehicle in association with that 
person near the partial image. 
[0019] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) computes distances 
between the vehicle and obstacles included in the 
image sensed by the image sensing device (3) on the 
basis of that image and the detection result from the dis- 
tance detection means (2), displays a partial image 
which represents the obstacle with the shortest com- 
puted distance to the vehicle on the display (8), and also 
displays the distance to the vehicle computed in associ- 
ation with that obstacle near the partial image. 
[0020] In a preferred embodiment of the display 
apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (1) computes distances 
between the vehicle and obstacles included in the over- 
all image sensed by the image sensing device (3) on the 
basis of that overall image and the detection result from 
the distance detection means (2), extracts a partial 
image that represents a corresponding obstacle when 
each of the computed distances is larger than a first pre- 
determined distance, executes a predetermined image 
processing for enlarging the extracted partial image, 
and displays an image that has undergone the image 
processing on the display (8). 

[0021] In the aforementioned apparatus arrange- 
ment, 
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the apparatus preferably further comprises vehicle 
velocity detection means (24) for detecting a vehi- 
cle velocity of the vehicle, and the display control 
means (21) makes enlarged-scale display of the 
partial image easier with increasing vehicle velocity 
of the vehicle upon executing the predetermined 
image processing. 

More preferably, the display control means makes 
enlarged-scale display of the partial image easier 
by increasing the first predetermined distance with 
increasing vehicle velocity of the vehicle; or 
the apparatus further comprises turn state detec- 
tion means (25) for detecting a turn state of the 
vehicle, and the display control means (21) inhibits 
execution of enlarged-scale display of the partial 
image when the turn state detection means detects 
that the vehicle is turning. 

Alternatively, the apparatus preferably further com- 
prises turn state detection means (25) for detecting 
a turn state of the vehicle, and when the turn state 
detection means detects that the vehicle is turning 
and it is determined that an obstacle located at a 
position closer than the first predetermined dis- 
tance is approaching along an estimated route of 
the vehicle, the display control means (21) makes 
enlarged-scale display of a partial image which rep- 
resents that obstacle, and when the turn state 
detection means detects that the vehicle is not turn- 
ing, the display control means inhibits execution of 
enlarged-scale display of the partial image. 
Alternatively, the first predetermined distance is 
preferably set at a possible irradiation distance of 
head lamps of the vehicle in the display control 
means. 

Alternatively, the display control means preferably 
also displays a symbol mark which informs a pas- 
senger of enlarged-scale display when the partial 
image is displayed in an enlarged scale. 
Alternatively, the apparatus preferably further com- 
prises maneuvering environment detection means 
(31 - 33) for detecting a maneuvering environment 
of the vehicle, and the display control means (21) 
corrects the first predetermined distance in accord- 
ance with the maneuvering environment detected 
by the maneuvering environment detection means. 

[0022] As a preferred embodiment adopted upon 
correcting the first predetermined distance in corre- 
spondence with the maneuvering environment, the fol- 
lowing processing is preferably done: 

the maneuvering environment detection means 
(31, 32) detects a value that pertains to visibility of 
a driver with respect to a maneuvering environment 
ahead of the vehicle as the maneuvering environ- 
ment, and the display control means corrects the 
first predetermined distance to increase as the 
detection result of the maneuvering environment 



10 
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20 



detection means indicates worse visibility. 
Alternatively, the maneuvering environment detec- 
tion means (31) detects a wiper operation speed of 
the vehicle as the maneuvering environment, and 
the display control means corrects the first prede- 
termined distance to increase with increasing wiper 
operation speed detected by the maneuvering envi- 
ronment detection means. 
Alternatively, the maneuvering environment detec- 
tion means (31) detects an ON/OFF state of fog 
lamps of the vehicle as the maneuvering environ- 
ment, and the display control means corrects the 
first predetermined distance to increase when the 
fog lamps are ON than a value when the fog lamps 
are OFF. 

[0023] In a preferred embodiment of the apparatus 
arrangement that displays the extracted partial image in 
an enlarged scale, preferably, 



the apparatus further comprises a manual switch 
(37) which can inhibit execution of enlarged-scale 
display by the display control means (21); or 
the display control means (21) superimposes an 
25 enlarged-scale image of the partial image on the 
overall image as the predetermined image process- 
ing. 

[0024] In a preferred embodiment of the display 
30 apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (21) computes distances 
between the vehicle and obstacles included in an over- 
all image sensed by the image sensing device (23) on 
the basis of that overall image and the detection result 
35 from the distance detection means (22), displays an 
image obtained by deforming a partial image that repre- 
sents a given obstacle on the display (28) when the 
computed distance to that obstacle is larger than a first 
predetermined distance, so as to make a driver easily 
40 recognize the presence of the corresponding obstacle, 
and displays the overall image on the display when the 
computed distance is smaller than the first predeter- 
mined distance. 

[0025] In a preferred embodiment of the display 
45 apparatus for a vehicle of the present invention, prefer- 
ably, the display control means (41) computes distances 
between the vehicle and obstacles included in an over- 
all image sensed by the image sensing device (43) on 
the basis of that overall image and the detection result 
so from the distance detection means (42), displays the 
overall image on the display (48) when the computed 
distance is smaller than a second predetermined dis- 
tance, and stops display of the overall image on the dis- 
play when the computed distance is larger than the 
55 second predetermined distance. 

[0026] In this case, the following arrangement is 
preferably used: 
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the apparatus further comprises vehicle velocity 
detection means (45) for detecting a vehicle veloc- 
ity of the vehicle, and the display control means 
changes the second predetermined distance to 
increase with increasing vehicle velocity of the vehi- 
cle. 

Alternatively, the apparatus preferably further com- 
prises maneuvering environment detection means 
(46, 53) for detecting a maneuvering environment 
of the vehicle, and the display control means cor- 
rects the second predetermined distance in accord- 
ance with the maneuvering environment detected 
by the maneuvering environment detection means. 

[0027] When the second predetermined distance is 
corrected in correspondence with the maneuvering 
environment, the following arrangement is adopted 
more preferably: 

the maneuvering environment detection means 
(46, 53) detects a value that pertains to visibility of 
a driver with respect to a maneuvering environment 
ahead of the vehicle as the maneuvering environ- 
ment, and the display control means corrects the 
second predetermined distance to increase as the 
detection result of the maneuvering environment 
detection means indicates worse visibility. 

[0028] More specifically, preferably, 

the maneuvering environment detection means 
(46) detects a wiper operation speed of the vehicle 
as the maneuvering environment, and the display 
control means corrects the second predetermined 
distance to increase with increasing wiper opera- 
tion speed detected by the maneuvering environ- 
ment detection means; or 

the maneuvering environment detection means 
(53) detects an ON/OFF state of fog lamps of the 
vehicle as the maneuvering environment, and the 
display control means corrects the second prede- 
termined distance to increase when the fog lamps 
are ON than a value when the fog lamps are OFF 

[0029] Upon controlling whether or not the entire 
image is displayed on the display (48) in accordance 
with the comparison result between the computed dis- 
tance and the second predetermined distance, 

the apparatus preferably further comprises alert 
means (50) for informing a passenger of start of dis- 
play by means of a voice message prior to display 
of the overall image. 

[0030] In this case, more preferably, 

the display control means corrects a delay time 
from an alert generation timing by the alert means 



until the start of display of the overall image on the 
display to be a time which decreases with decreas- 
ing distance computed for the obstacle; or 
the apparatus further comprises vehicle velocity 

5 detection means (45) for detecting a vehicle veloc- 
ity of the vehicle, and the display control means cor- 
rects a delay time from an alert generation timing by 
the alert means until the start of display of the over- 
all image on the display to be a time which 

10 decreases with increasing vehicle velocity of the 
vehicle. 

[0031] According to the present invention, even 
when a plurality of obstacles are found, the driver can 
15 easily recognize an obstacle which he or she must 
watch out. 

[0032] Other features and advantages of the 
present invention will be apparent from the following 
description taken in conjunction with the accompanying 
20 drawings, in which like reference characters designate 
the same or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 [0033] 

Fig. 1 is a block diagram of a display apparatus for 
a vehicle according to the first embodiment of the 
present invention; 
30 Fig. 2 is a flow chart of a display control processing 
by the display apparatus for a vehicle according to 
the first embodiment; 

Fig. 3 shows an example of an image sensed by an 
infrared ray camera (normal display mode); 
35 Fig. 4 is a top view of the detection range of an 
obstacle sensor 2; 

Fig. 5 shows a display example in the first embodi- 
ment; 

Fig. 6 is a flow chart of a display control processing 
40 by a display apparatus for a vehicle according to the 
second embodiment of the present invention; 
Fig. 7 shows a display example in the second 
embodiment (when a plurality of obstacles are 
present); 

45 Fig. 8 shows a display example in the second 
embodiment (when one obstacle is present); 
Fig. 9 shows a display example of an obstacle dis- 
played in an enlarged scale in the third embodiment 
of the present invention; 

so Fig. 10 is a flow chart showing only steps, which are 
different from the second embodiment, of the dis- 
play control processing by a display apparatus for a 
vehicle according to the fourth embodiment of the 
present invention; 

55 Fig. 1 1 shows the radiation temperature range of a 
person to be set in a display controller; 
Figs. 12A and 12B show display examples in the 
fourth embodiment; 
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Fig. 1 3 is a flow chart showing only steps, which are 
different from the second embodiment, of the dis- 
play control processing by a display apparatus for a 
vehicle according to the fifth embodiment of the 
present invention; 5 
Fig. 14 shows a display example in the fifth embod- 
iment; 

Figs. 15A and 15B show display examples in a 
modification of the fifth embodiment; 
Fig. 1 6 is a block diagram of a display apparatus for w 
a vehicle according to the sixth embodiment of the 
present invention; 

Fig. 17 is a flow chart of a display control process- 
ing by the display apparatus for a vehicle according 
to the sixth embodiment; 15 
Fig. 18 shows an example of an infrared ray image 
sensed by an infrared ray camera 23; 
Fig. 19 shows an example of a partial image includ- 
ing obstacles, which is displayed in an enlarged 
scale, of the infrared ray image; 20 
Fig. 20 shows an example of a partial image includ- 
ing obstacles, which is displayed in an enlarged 
scale, of the infrared ray image; 
Fig. 21 shows an example of a partial image includ- 
ing an obstacle, which is emphasized and displayed 25 
in an enlarged scale, of the infrared ray image; 
Fig. 22 shows an example of a reduced-scale 
image of an infrared ray image, and an enlarged- 
scale image of an obstacle, which are displayed at 
the same time; 30 
Fig. 23 is a flow chart of a display control process- 
ing by a display apparatus for a vehicle according to 
the seventh embodiment of the present invention; 
Fig. 24 is a flow chart of a display control process- 
ing by a display apparatus for a vehicle according to 35 
the eighth embodiment of the present invention; 
Fig. 25 is a block diagram of a display apparatus for 
a vehicle according to the ninth embodiment of the 
present invention; 

Fig. 26 is a flow chart of a display control process- 40 
ing by the display apparatus for a vehicle according 
to the ninth embodiment; 

Fig. 27 is a flow chart of a display control process- 
ing by a display apparatus for a vehicle according to 
the 10th embodiment of the present invention; 45 
Fig. 28 shows an example of an image sensed by a 
visible light camera 47, and a coordinate axis set for 
that image; 

Fig. 29 shows an example of a contrast detection fil- 
ter; 50 
Fig. 30 is a flow chart of a display control process- 
ing by a display apparatus for a vehicle according to 
the 1 1th embodiment of the present invention; and 
Figs. 31 A and 31 B are views for explaining a 
method of detecting the driver's face direction by a 55 
head/face position detection unit 44 in the 11th 
embodiment. 



DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0034] Preferred embodiments to which a display 
apparatus for a vehicle according to the present inven- 
tion is applied to an automobile as a representative 
vehicle will be described in detail hereinafter with refer- 
ence to the accompanying drawings. 

[First Embodiment] 

[0035] Fig. 1 is a block diagram of a display appara- 
tus for a vehicle according to the first embodiment of the 
present invention. 

[0036] Referring to Fig. 1, reference numeral 2 
denotes an obstacle sensor such as a CCD (Charge 
Coupled Device) camera, laser radar, or millimeter- 
wave radar, or the like for detecting the distance to an 
obstacle ahead of the vehicle by a general method. Ref- 
erence numeral 3 denotes an infrared light (infrared ray) 
camera using an environment ahead of the vehicle 
using infrared rays. 

[0037] Reference numeral 4 denotes a GPS (global 
positioning system) sensor for externally receiving a 
GPS signal used upon computing the current position of 
the vehicle. Reference numeral 5 denotes an atmos- 
pheric temperature sensor for detecting the tempera- 
ture outside the passenger room of the vehicle. 
Reference numeral 6 denotes a map database from 
which data such as road map information or the like can 
be read out. 

[0038] Reference numeral 8 denotes a display such 
as a liquid crystal display, head-up display, or the like for 
displaying an image sensed by the infrared light camera 
3 (to be referred to as a sensed image hereinafter). 
Note that the display 8 is preferably laid out at a position 
(e.g., near the central position of a dashboard, or the 
like) in front of the driver's seat of the vehicle where the 
driver can easily see the displayed image without 
largely moving the line of sight when he or she looks for- 
ward. 

[0039] Reference numeral 11 denotes a mode 
select switch which allows the operator to select a dis- 
play mode of a sensed image on the display 8. The dis- 
play mode that can be selected by this switch includes a 
manual mode which directly displays a sensed image 
output from the infrared light camera 3 on the display 8 
without any modifications while the operator holds a 
manual ON/OFF switch 13 ON, and an auto mode 
which displays an image processed by a display control 
processing (to be described later) on the display 8, 
when the auto mode is selected by the mode select 
switch 11. 

[0040] Reference numeral 12 denotes a power 
switch which allows the operator to turn on/off power 
supply to a display controller 1 . 
[0041] The manual ON/OFF switch 13 is used to 
display the sensed image output from the infrared light 
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camera 3 on the display 8 without any modifications 
while the operator holds this switch ON when the man- 
ual mode is selected by the mode select switch 1 1 . 
[0042] The display controller 1 controls display of 
the sensed image output from the infrared light camera 
3 on the display 8 on the basis of output signals (data 
corresponding to the output signals) from the respective 
sensors, and the detected operation states of the oper- 
ation switches, as will be described in detail later. 
[0043] The display control processing of the display 
controller 1 is executed by a CPU 101 in accordance 
with software pre-stored in a ROM 103 or the like while 
using a RAM 102 as a work area. 
[0044] An example of the display control processing 
done by the display controller 1 in this embodiment will 
be described below with reference to Figs. 2 to 5. 
[0045] Fig. 2 is a flow chart of the display control 
processing by the display apparatus for a vehicle 
according to the first embodiment. Fig. 2 shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
power switch 12 is ON. 

[0046] Step S1 in Fig. 2: The CPU 101 detects the 
operation state of the mode select switch 1 1 to check if 
the manual mode is selected. If YES (manual mode) In 
step S1, the flow advances to step S2; if NO (auto 
mode) in step S1 , the flow advances to step S5. 
[0047] Steps S2 to S4: In the current state wherein 
the manual mode is selected, the CPU 101 checks if the 
manual ON/OFF switch 13 is ON. If NO (switch 13: 
OFF) in step S2, the CPU 101 turns off display of the 
sensed image on the display 8 (step S3); if YES (switch 
13: ON) in step S2, the CPU 101 starts or continues dis- 
play of an image sensed by the infrared ray camera 3 
(i.e., an infrared ray image which does not undergo any 
processing in the auto mode to be described later: see 
a display example shown in Fig. 3) (step S4). 
[0048] Step S5: Since the auto mode is selected, 
the CPU 101 loads data corresponding to the output 
signal from the obstacle sensor in this step. 
[0049] Step S6: The CPU 1 01 determines an obsta- 
cle located at a position closest to the vehicle on the 
basis of the data which is loaded in step S5 and corre- 
sponds to the output signal from the obstacle sensor 2, 
and the sensed image output from the infrared light sen- 
sor 3. 

[0050] An example of the method of detecting the 
closest obstacle in step S6 will be explained below. 
[0051] Fig. 4 is a top view of the detection range of 
the obstacle sensor 2, and shows a case wherein the 
detection range of the sensor 2 includes obstacle A 
(another leading vehicle) and obstacle B (pedestrian), 
which correspond to the sensed image output from the 
infrared light camera 3 shown in Fig. 3. 
[0052] In this embodiment, the detection results of 
obstacles A and B by the obstacle sensor 2 are 
obtained as a distance Da between the vehicle and 
obstacle A and an angle 6a (= 0) obstacle A makes with 



the central axis, and a distance Db between the vehicle 
and obstacle B and an angle 6b obstacle B makes with 
the central axis. 

[0053] Therefore, when the position relationship 

5 between the central axis of the detection range of the 
obstacle sensor 2, and the coordinate axis of the image 
sensing surface (field angle) of the infrared light camera 
3 is determined in advance in correspondence with 
each other, the distance D to an obstacle output from 

10 the obstacle sensor 2 is counted as the number of lines 
(the number of pixels) from the lower side of the image 
sensing surface, and the angle 6 is counted as the 
number of pixels from the central line in the horizontal 
direction of the image sensing surface, as shown in Fig. 

15 3. As a result, it can be determined that a radiation heat 
source including a position (pixel) in the image sensing 
surface, which is specified by these counts, corre- 
sponds to the entire shape of an obstacle detected by 
the obstacle sensor 2. Therefore, if such processing is 

20 done in step S6, the display controller 1 can recognize 
an obstacle included in the sensed image of the infrared 
light camera 3 and the distance D to that obstacle in 
association with each other. 

[0054] Step S7: Only an image of an obstacle 
25 located at a position closest to the vehicle (nearby posi- 
tion) detected in step S6 is displayed on the display 8 
(like a display example shown in Fig. 5 with respect to 
the sensed image of the infrared light camera 2 shown 
in Fig. 3). 

30 [0055] As an example of a processing in this step, 
display colors of surrounding pixels other than an image 
corresponding to the obstacle located at the position 
closest to the vehicle of an image signal for one frame 
that includes the image of the obstacle are converted 

35 into black (expressed by white in Fig. 5) before the 
image is displayed. 

[0056] At this time, as a method of identifying a 
pixel group that form the obstacle detected in step S6 
from a plurality of pixels (pixel values) that form an 

40 image signal for one frame, a radiation heat source (cor- 
responding to a person in front of the vehicle in Fig. 3) 
corresponding to the obstacle detected by the infrared 
ray camera 3 normally has a higher temperature than 
the ambient temperature of the obstacle, and the tem- 

45 perature difference within that radiation heat source is 
smaller than that around the radiation heat source. 
Hence, by detecting this temperature difference, the 
pixel group (radiation heat source) that form the obsta- 
cle (an obstacle closest to the vehicle) can be easily 

so identified from the pixel group around that obstacle. 
[0057] As described above, according to this 
embodiment, since only the image of the obstacle clos- 
est to the vehicle is displayed on the display 8, even 
when the sensed image output from the infrared light 

55 camera 3 actually includes a plurality of obstacles, the 
driver can easily recognize the presence of the obsta- 
cle, which he or she must watch out most upon driving 
the vehicle. 
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[0058] In this embodiment, the aforementioned dis- 
play control is done in the auto mode. Alternatively, the 
processing for emphasizing and displaying the obstacle 
located at the position closest to the vehicle may be 
done when an image is displayed on the display 8 in the 
manual mode. 

[Second Embodiment] 

[0059] The second embodiment, which is based on 
the aforementioned display apparatus for a vehicle 
according to the first embodiment, will be explained 
below. In the following description, an explanation of the 
same arrangement as that in the first embodiment will 
be omitted, and a characteristic feature of this embodi- 
ment will be mainly explained. 
[0060] Fig. 6 is a flow chart of a display control 
processing by the display apparatus for a vehicle 
according to the second embodiment, and shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
power switch 1 1 is ON. 

[0061] Referring to Fig. 6, the processing in steps 
. S1 1 to S1 5 are the same as those in steps S1 to S5 in 
the first embodiment (Fig. 2). 

[0062] Step S16: The CPU 101 checks if data cor- 
responding to the output signal from the obstacle sen- 
sor 2, which is loaded in step S15, includes data which 
represent distances D and angles 9 associated with a 
plurality of obstacles. If YES in step S16 (if a plurality of 
obstacles are present), the flow advances to step S17; 
if NO in step S16 (if one obstacle is present), the flow 
advances to step S19. 

[0063] Step S1 7: Since it is determined in step S16 
that there are a plurality of obstacles, in this step the 
CPU 101 selects data corresponding to an output sig- 
nal, which represents the shortest distance to the vehi- 
cle, of those of the plurality of obstacles detected by the 
obstacle sensor 2, and associates the selected data 
with an image (pixel group) of the obstacle correspond- 
ing to that data, which is included in a sensed image 
output from the infrared light camera 3, as in the 
processing in step S6 in the first embodiment (Fig. 2). 
[0064] Step S18: Only the image of the obstacle 
associated in step S17 is emphasized and displayed on 
the display 8, as shown in a display example in Fig. 7, 
and the flow returns. 

[0065] As examples of such emphasis display 
mode, a mode for changing the contrast of the image of 
the obstacle of interest from that of the surrounding 
image, a mode for increasing the luminance, a mode for 
changing the display color, a mode for outline-displaying 
the image of the obstacle of interest, or the like may be 
used. On the other hand, the method of recognizing a 
pixel group that form the image of the associated clos- 
est obstacle in the sensed image can be the same as 
that in step S7 in the first embodiment (Fig. 2). 
[0066] Step S1 9: Since it is determined in step S16 



that one obstacle is present, in this step the CPU 101 
displays the sensed image output from the infrared light 
camera 3 on the display 8 without any modification, as 
shown in a display example of Fig. 8, and the flow 
5 returns. 

[0067] As described above, according to this 
embodiment, since the display mode of the image of the 
obstacle closest to the vehicle to be displayed on the 
display 8 is emphasized, even when the sensed image 
10 output from the infrared light camera 3 actually includes 
a plurality of obstacles, the driver can easily recognize 
the presence of the obstacle, which he or she must 
watch out most upon driving the vehicle. 

15 [Third Embodiment] 

[0068] The third embodiment, which is based on the 
aforementioned display apparatus for a vehicle accord- 
ing to the second embodiment, will be explained below. 
20 In the following description, an explanation of the same 
arrangement as that in the second embodiment will be 
omitted, and a characteristic feature of this embodiment 
will be mainly explained. 

[0069] In this embodiment, the processing in step 
25 S18 is different from that in the second embodiment 
(Fig. 6). 

[0070] More specifically, this embodiment com- 
putes a zoom ratio at which an image of the associated, 
closest obstacle can be displayed to the full length or 

30 width of the display 8, and extracts a partial image cor- 
responding to that zoom ratio from the sensed image 
output from the infrared light camera 3. Then, an 
enlarged-scale image which is obtained by executing a 
general digital zoom processing of the extracted partial 

35 image is displayed on the display 8 (as shown in a dis- 
play example of Fig. 9 with respect to the sensed image 
from the infrared light camera 2 shown in, e.g., Fig. 3). 
[0071] As described above, according to this 
embodiment, since the image of the obstacle closest to 

40 the vehicle is displayed on the display 8 in an enlarged 
scale, even when the sensed image output from the 
infrared light camera 3 actually includes a plurality of 
obstacles, the driver can easily recognize the presence 
of the obstacle, which he or she must watch out most 

45 upon driving the vehicle. 

[Fourth Embodiment] 

[0072] The fourth embodiment, which is based on 
50 the aforementioned display apparatus for a vehicle 
according to the second embodiment, will be explained 
below. In the following description, an explanation of the 
same arrangement as that in the second embodiment 
will be omitted, and a characteristic feature of this 
55 embodiment will be mainly explained. 

[0073] Fig. 10 is a flow chart showing only steps, 
which are different from the second embodiment, of the 
display control processing by a display apparatus for a 
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vehicle according to the fourth embodiment of the 
present invention. 

[0074] Step S27 in Fig. 10: Since it is determined in 
step S16 that there are a plurality of obstacles, images 
of persons are extracted from the images of those 5 
obstacles. As an example of a method of extracting an 
image portion of a person, since it is known that radia- 
tion heat of a person sensed by the infrared light cam- 
era normally falls within a predetermined temperature 
range, a heat source, the detected (sensed) radiation 10 
temperature of which falls within the predetermined 
temperature range, can be selected from the sensed 
image output from the infrared light camera 3. 
[0075] Also, it is known that the temperature range 
of radiation heat of a normal person increases with 15 
increasing atmospheric temperature. Hence, a plurality 
of temperature ranges are prepared in advance, as 
shown in Fig. 11, one of these temperature ranges is 
selected in accordance with the detection result from 
the atmospheric temperature sensor 5, a timepiece 20 
equipped in the display controller 1 or the vehicle, or cal- 
endar or time information acquired from a GPS signal 
using the GPS sensor 4, and the selected temperature 
range can be used in discrimination in step S27. 
[0076] Step S28: One of the extracted image por- 25 
tions of the persons, which is closest to the vehicle, is 
determined in the same manner as in the processing in 
step S17 in the second embodiment (Fig. 6), i.e., that in 
step S6 in the first embodiment (Fig. 2). 
[0077] Step S29: Only the image of the person clos- 30 
est to the vehicle, which is extracted in step S28, is 
emphasized and displayed on the display 8, as shown in 
a display example in Fig. 12A, and the flow returns. As 
examples of emphasis display modes, a mode for 
increasing the luminance, a mode for changing the dis- 35 
play color, a mode for outline-displaying the image of 
the obstacle of interest, or the like may be used as in 
stepS 18. 

[0078] In this embodiment, only the image of a per- 
son closest to the vehicle is emphasized and displayed. 40 
Alternatively, when a predetermined number of images 
of persons may be selected in ascending order of dis- 
tance D to the vehicle on the basis of the detection 
result of the obstacle sensor 2 in step S28, and the 
selected images may be emphasized and displayed in 45 
step S29, not only the closest person but also a prede- 
termined number of persons close to that person may 
be emphasized and displayed, as shown in a display 
example of Fig. 12B. 

[0079] As described above, according to this so 
embodiment, since the image of the obstacle closest to 
the vehicle is emphasized and displayed on the display 
8, even when the sensed image output from the infrared 
light camera 3 actually includes a plurality of obstacles, 
the driver can easily recognize the presence of the per- 55 
son to be protected, which he or she must watch out 
most upon driving the vehicle. 



[Fifth Embodiment] 

[0080] The fifth embodiment, which is based on the 
display apparatus for a vehicle according to the second 
and fourth embodiments mentioned above, will be 
described below. In the following description, an expla- 
nation of the same arrangement as that in the second 
embodiment will be omitted, and a characteristic feature 
of this embodiment will be mainly explained. 
[0081] Fig. 13 is a flow chart showing only steps, 
which are different from the second embodiment, of the 
display control processing by a display apparatus for a 
vehicle according to the fifth embodiment of the present 
invention. 

[0082] Step S37 in Fig. 13: Since it is determined in 
step S16 that there are a plurality of obstacles, images 
of persons are extracted from the images of those 
obstacles as in step S27 in the fourth embodiment (Fig. 
10). 

[0083] Step S38: One of the extracted persons, who 
is closest to the vehicle, is determined as in step S28 in 
the fourth embodiment (Fig. 10). The sensed image 
from the infrared light camera 3 is displayed on the dis- 
play 8, and the distance value to the vehicle as the 
detection result of the obstacle sensor 2, which is 
referred to upon determining the person closest to the 
vehicle, is superimposed near the image of that person 
included in the displayed image, as shown in a display 
example of Fig. 14. 

[0084] As described above, since the distance to 
the person closest to the vehicle is superimposed on the 
display 8 as a numerical value, even when the sensed 
image output from the infrared light camera 3 actually 
includes a plurality of obstacles, the driver can easily 
recognize the presence of the person to be protected, 
which he or she must watch out most upon driving the 
vehicle. 

Modification of Fifth Embodiment) 

[0085] In the fifth embodiment described above, the 
distance value is superimposed near only the image of 
the person upon displaying the sensed image from the 
infrared light camera 3 on the display 8. However, this 
embodiment is not limited to persons, but may superim- 
pose the distance value near an obstacle closest to the 
vehicle. 

[0086] Therefore, unlike in the fifth embodiment, the 
processing in step S37 of extracting image portions of 
persons from the sensed image from the infrared light 
camera 3 need not be executed, Also, in step S38 an 
obstacle closest to the vehicle is determined, the 
sensed image from the infrared light camera 3 is dis- 
played on the display 8, and the distance value to the 
vehicle as the detection result of the obstacle sensor 2, 
which is referred to upon determining the obstacle clos- 
est to the vehicle, is superimposed near the image of 
that obstacle included in the displayed image, as shown 
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in a display example of Fig. 15A or 15B. 
[0087] As described above, according to this modi- 
fication, since the distance to the obstacle closest to the 
vehicle is superimposed displayed on the display 8 as a 
numerical value, even when the sensed image output 
from the infrared light camera 3 actually includes a plu- 
rality of obstacles, the driver can easily recognize the 
presence of the obstacle, which he or she must watch 
out most upon driving the vehicle. 

[Sixth Embodiment] 

[0088] Fig. 16 is a block diagram of a display appa- 
ratus for a vehicle according to the sixth embodiment of 
the present invention. 

[0089] Referring to Fig. 16, reference numeral 22 
denotes an obstacle sensor such as a CCD (Charge 
Coupled Device) camera, laser radar, millimeter-wave 
radar, or the like for detecting the distance to an obsta- 
cle ahead of the vehicle by a known method. Reference 
numeral 23 denotes an infrared light (infrared ray) cam- 
era for sensing an image of an environment ahead of 
the vehicle using infrared rays. 
[0090] Reference numeral 24 denotes a vehicle 
velocity sensor for detecting the vehicle velocity as a 
travel state of the vehicle. Reference numeral 25 
denotes a steering angle sensor for detecting the steer- 
ing angle of the vehicle. 

[0091] Reference numeral 28 denotes a display 
such as a liquid crystal display, head-up display, or the 
like for displaying an image sensed by the infrared light 
camera 23 (to be referred to as an infrared ray image 
hereinafter). Note that the display 28 is preferably laid 
out at a position (e.g., near the central position of a 
dashboard) in front of the driver's seat of the vehicle 
where the driver can easily see the displayed image 
without largely moving the line of sight when he or she 
looks forward. 

[0092] Reference numeral 31 denotes a wiper 
switch with which the driver can start/stop wipers pro- 
vided to the vehicle, and can set its operation state such 
as an operation speed (period) or the like. Reference 
numeral 32 denotes a fog lamp switch with which the 
driver can turn on/off fog lamps provided to the vehicle. 
Reference numeral 33 denotes a head lamp switch with 
which the driver can turn on/off head lamps provided to 
the vehicle. 

[0093] Reference numeral 34 denotes a power 
switch which allows the operator (driver) to turn on/off 
power supply to a display controller 21. Reference 
numeral 35 denotes a mode select switch which allows 
the operator to select a desired display mode of a 
sensed image on the display 28. The display mode that 
can be selected by this switch includes a manual mode 
which displays an infrared light image on the display 28 
while the operator holds a manual ON/OFF switch 37 
ON, and an auto mode which displays an infrared light 
image on the display 28 in accordance with a display 



83 076A2 18 

control processing (to be described later) when the auto 
mode is selected by the mode select switch 35. 
[0094] Reference numeral 36 denotes a zoom 
ON/OFF switch which allows the driver to select 
5 whether or not an image of an obstacle included in the 
infrared ray image is displayed in an enlarged scale 
(zoom mode) when the auto mode is selected by the 
mode select switch 35. 

[0095] The manual ON/OFF switch 37 is used to 
10 display the infrared ray sensed image on the display 28 
while the operator holds this switch ON when the man- 
ual mode is selected by the mode select switch 35. 
[0096] The display controller 21 controls display of 
the infrared ray image on the display 28 on the basis of 
15 output signals (data corresponding to the output sig- 
nals) from the respective sensors, and the detected 
operation states of the operation switches (to be 
described in detail later). 

[0097] The display control processing of the display 
20 controller 21 is executed by a CPU 101 in accordance 
with software pre-stored in a ROM 103 or the like while 
using a RAM 102 as a work area. 
[0098] An example of the display control processing 
done by the display controller 21 in this embodiment will 
25 be described below. 

[0099] Fig. 17 is a flow chart of the display control 
processing by the display apparatus for a vehicle 
according to the sixth embodiment. Fig. 17 shows the 
sequence of software executed by the CPU 101 while 
30 the ignition key switch of the vehicle is ON, and the 
power switch 34 is ON. 

[0100] Step S101 in Fig. 17: The CPU 101 checks 
by detecting the operation state of the mode select 
switch 35 if the auto mode is selected. If NO in step 
35 S1 01 (manual mode), the flow advances to step S102; if 
YES in step S101 (auto mode), the flow advances to 
step S104. 

[0101] Step S102: Since the manual mode is 
selected, the CPU 101 checks if the manual ON/OFF 

40 switch 37 is ON. If YES in step S102 (switch 37: ON), 
the flow advances to step S1 1 1 to start or continue dis- 
play of the infrared ray image; if NO in step S102 (switch 
37: OFF), the flow advances to step S103 to stop dis- 
play of the infrared ray image. 

45 [0102] Step S103: The CPU 103 stops display of 
the infrared ray image on the display 28, or continues a 
stop state. 

[0103] Step S104: Since the auto mode is selected, 
the CPU 101 checks if the zoom ON/OFF switch 36 is 

50 ON. If YES in step S104 (switch 36: ON), the flow 
advances to step S105; if NO in step S104 (switch 36: 
OFF), the flow advances to step S1 1 1 to start or con- 
tinue display of the infrared ray image, which is sensed 
by the infrared light camera 23 and is not modified. 

55 [0104] Step S105: The CPU 101 loads data corre- 
sponding to the output signals from the respective sen- 
sors and signals representing the operation states of 
the respective switches, which have been explained 
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above with reference to Fig. 16. 
[0105] Step S106: The CPU 101 checks if the data 
corresponding to the output signals from the vehicle 
velocity sensor 24 and steering angle sensor 25 
acquired in step S105 respectively represent a vehicle 5 
velocity larger than a predetermined velocity and a 
steering angle larger than a predetermined angle, so as 
to check if the vehicle is traveling while making a turn. If 
YES in step S106 (the vehicle is turning), the flow 
advances to step S1 1 1 to start or continue direct display w 
of the infrared ray image sensed by the infrared light 
camera 23; if NO in step S106 (the vehicle is not turn- 
ing), the flow advances to step S107. 
[0106] The reason why enlarged-scale display of an 
obstacle is inhibited while the vehicle is turning is that 15 
turning requires the driver to processing more informa- 
tion and take more actions than in straight travel, and 
exerts a heavier mental load on driving. Hence, the 
driver must be prevented from taking fright or being dis- 
tracted if enlarged-scale display starts in such situation. 20 
[0107] Step S107: The CPU 101 sets a display 
threshold value Th, which serves as a reference in 
determining if zoom display is made upon displaying an 
infrared ray image on the display 28, in correspondence 
with the magnitude of the data corresponding to the out- 25 
put signal from the vehicle velocity sensor 24 acquired 
in step S105, with reference to a map (table) pre-stored 
in the ROM 103. 

[0108] As an example of the characteristics of the 
display threshold value Th stored in this map, a thresh- 30 
old value which increases with increasing vehicle veloc- 
ity can be set. 

[0109] Step S108: The CPU 101 checks the pres- 
ence/absence of an obstacle ahead of the vehicle on 
the basis of the data corresponding to the output signal 35 
from the obstacle sensor 22 acquired in step S105. If 
YES in step S108 (presence of an obstacle), the flow 
advances to step S109; if NO in step S108 (absence of 
an obstacle), the flow advances to step S1 1 1 to start or 
continue direct display of the infrared ray image sensed 40 
by the infrared light camera 23. 
[0110] Step S109: The CPU 101 acquires the loca- 
tion of the obstacle and distance D between that obsta- 
cle and the vehicle on the basis of the data 
corresponding to the output signal from the obstacle 45 
sensor 22 acquired in step S105, and checks if the dis- 
tance D is larger than the display threshold value Th set 
in step S107. If YES in step S109 (D > Th), the flow 
advances to step S110 to make enlarged-scale display 
since the obstacle is faraway from the vehicle and it is so 
hard for the driver to visually recognize the display con- 
tents; if NO in step S109 (D < TH), the flow advances to 
step S1 1 1 to start or continue direct display of the infra- 
red ray image sensed by the infrared light camera 23. 
[0111] Step S111: The CPU 101 starts or continues 55 
direct display of an infrared ray image (e.g., an image 
from which an inner frame is removed from that shown 
in Fig. 18) sensed by the infrared light camera 23. 



[0112] Step S110: The CPU 101 associates the 
obstacle detected by the obstacle sensor 22 with the 
image of the obstacle contained in the infrared ray 
image sensed by the infrared light camera 23, com- 
putes a zoom ratio at which the image of that obstacle 
can be displayed to the full length or width of the display, 
and extracts a partial image corresponding to that zoom 
ratio from the sensed image of the infrared light camera 
23. 

[0113] Then, the CPU 101 executes a general dig- 
ital zoom processing of the extracted partial image to 
obtain an enlarged-scale image (as shown in, e.g., a 
display example in Fig. 19 with respect to the sensed 
image of the infrared light camera 23 shown in Fig. 18), 
and displays that image on the display 28. 
[0114] Upon displaying the enlarged-scale image 
on the display 28 in this step, a symbol indicating an 
enlarged-scale image is preferably superimposed, as 
shown in Fig. 19, so as to inform the driver of that state. 
[0115] An example of the method of associating the 
obstacle detected by the obstacle sensor 22 and the 
image of the obstacle contained in the infrared ray 
image sensed by the infrared light camera 23 is the 
same as that in the first embodiment mentioned above 
with reference to Figs. 3 and 4. 
[0116] That is, in this embodiment, when the posi- 
tional relationship between the central axis of the detec- 
tion range of the obstacle sensor 22 and the coordinate 
axis of the image sensing surface (field angle) of the 
infrared light camera 23 is determined in advance in 
correspondence with each other, the distance D to the 
obstacle output from the obstacle sensor 22 can be 
counted as the number of lines (the number of pixels) 
from the lower side of the image sensing surface, and 
the angle 9 is counted as the number of pixels from the 
central line in the horizontal direction of the image sens- 
ing surface, as shown in Fig. 3. As a result, it can be 
determined that a radiation heat source including a 
position (pixel) in the image sensing surface, which is 
specified by these counts, corresponds to the entire 
shape of an obstacle detected by the obstacle sensor 
22. 

[0117] Therefore, when such processing is done in 
step S110, the display controller 21 can recognize the 
partial image of the obstacle contained in the sensed 
image from the infrared light camera 23, and the dis- 
tance D to that obstacle in association with each other. 
[0118] If it is determined based on the data corre- 
sponding to the output signal from the obstacle sensor 
22 acquired in step S105 in the processing in step S1 10 
that there are a plurality of obstacles, the smallest one 
of a plurality of distances D of these obstacles is prefer- 
ably superimposed on the enlarged-scale display of that 
obstacle, as shown in Fig. 20, so as to make the driver 
recognize the distance between that obstacle and the 
vehicle. 

[0119] In the processing in step S1 10, the distance 
D included in the data corresponding to the output sig- 
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nal from the obstacle sensor 22 is compared with the 
display threshold value Th. In a specific maneuvering 
environment such as fog, rain, or the like, the distance D 
is corrected to be a larger value with increasing opera- 
tion speed represented by the operation state of the 
wiper switch 31 (i.e., with decreasing operation period) 
or to be a larger value than that in the OFF state of the 
fog lamp switch 32 when the operation state of the fog 
lamp switch 32 indicates the ON state, prior to compar- 
ison with the display threshold value Th. 
[0120] Conversely, the display threshold value Th 
set in step S107 may be corrected to decrease with 
increasing operation speed represented by the opera- 
tion state of the wiper switch 31 (i.e., with decreasing 
operation period) or to be a value below that in the OFF 
state of the fog lamp switch 32 when the operation state 
of the fog lamp switch 32 indicates the ON state, prior to 
comparison with the display threshold value Th, so that 
the image of the detected obstacle is displayed in an 
enlarged scale at a relatively earlier timing. 
[0121] Likewise, in the processing in step S109, the 
display threshold value Th set in accordance with the 
detected vehicle velocity in step S107 is compared with 
the detected distance D. By contrast, the display thresh- 
old value Th may be set as a constant, and the distance 
D may be corrected in accordance with the vehicle 
velocity. 

[0122] In place of the enlarged-scale display shown 
in Fig. 19, only a deformed image of an obstacle closest 
to the vehicle may be displayed, as shown in Fig. 21 , so 
that the driver can recognize the shape (outline) of that 
obstacle, or a reduced-scale image of original infrared 
ray image A that contains obstacle B and an enlarged- 
scale image of obstacle B may be simultaneously dis- 
played, as shown in Fig. 22. Since image processing 
that implement these display modes are the state-of- 
the-art techniques, a detailed description thereof will be 
omitted in this embodiment. 

[0123] According to this embodiment, when the 
auto mode is selected, and the zoom ON/OFF switch 36 
is ON, if the distance D between the vehicle and a given 
obstacle is larger than the display threshold value Th, 
since the image of the obstacle contained in the infrared 
ray image is displayed in an enlarged scale, the driver 
can easily recognize the obstacle far away from the 
vehicle. 

[0124] The display threshold value Th used upon 
determining if enlarged-scale display is made is set to 
increase with increasing detected vehicle velocity of the 
vehicle. In this manner, the driver can recognize the 
obstacle present ahead of the vehicle early in high- 
speed cruise, and can assure a sufficient time to dodge 
that obstacle. 

[0125] Also, since the display threshold value Th is 
corrected depending on a maneuvering environment 
estimated in accordance with the operation state of the 
wipers or the ON/OFF state of the fog lamps prior to 
determination of enlarged-scale display, enlarged-scale 



display starts at an earlier timing in rough weather that 
exerts a heavier mental load on driving than on fine 
weather, and can adequately assist driver's vision. 
[0126] In this embodiment, the aforementioned dis- 
5 play control is done in the auto mode. Alternatively, the 
processing for displaying the image of an obstacle in an 
enlarged scale may be made when the image is dis- 
played on the display 28 in the manual mode. 

10 [Seventh Embodiment] 

[0127] The seventh embodiment, which is based on 
the aforementioned display apparatus for a vehicle 
. according to the sixth embodiment, will be described 

15 below. In the following description, an explanation of the 
same arrangement as that in the sixth embodiment will 
be omitted, and a characteristic feature of this embodi- 
ment will be mainly explained. 
[0128] Fig. 23 is a flow chart of a display control 

20 processing by the display apparatus for a vehicle 
according to the seventh embodiment, and shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
power switch 34 is ON. 

25 [0129] Steps S121 to S125 in Fig. 23: The same 
processing as in steps S101 to S105 in the sixth embod- 
iment (Fig. 17) are executed, and if it is determined in 
step S124 that the zoom ON/OFF switch is OFF, the 
flow advances to step S132. 

30 [0130] Step S126: The CPU 101 sets a display 
threshold value Th, which serves as a reference in 
determining if zoom display is made upon displaying an 
infrared ray image on the display 28, in correspondence 
with the operation state (operation speed/period) of the 

35 wipers and the ON/OFF state of the fog lamps, with ref- 
erence to a map (table) pre-stored in the ROM 103. 
[0131] For example, the display threshold value Th 
is set to decrease with increasing operation speed (i.e., 
decreasing operation period) represented by the opera- 

40 tion state of the wiper switch 31 acquired in step S125. 
Or the display threshold value Th is set to be a value 
below that of the OFF state of the fog lamp switch 32 
when the operation state of the fog lamp switch 32 indi- 
cates the ON state. 

45 [0132] Steps S127 and S128: The CPU 101 checks 
if an obstacle is present, and checks if the vehicle is 
turning if an obstacle is present, as in the processing in 
steps S108 and S106 in the sixth embodiment (Fig. 17). 
If the vehicle is not turning, the flow advances to step 

so S130. 

[0133] Step S129: Since the vehicle is turning, it is 
not preferable that the detected obstacle approaches 
the traveling route along which the vehicle is expected 
to travel. 

55 [0134] Hence, in this step the CPU 101 checks if 
the detected obstacle is moving in a direction away from 
the turning direction of the vehicle or stands still. If NO 
in step S129 (the obstacle relatively approaches the 
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vehicle), the flow advances to step S130 since the 
image of that obstacle should be displayed in an 
enlarged scale; if YES in step S129 (the obstacle is 
moving in a direction opposite to the traveling direction 
or stands still), the flow advances to step S132 to start 5 
or continue direct display of the infrared ray image 
sensed by the infrared light camera 23. 
[0135] As an example of the processing in this step, 
the CPU 101 estimates the traveling direction of the 
vehicle (predicted route along which the vehicle is 10 
expected to travel) on the basis of data representing the 
output signals from the vehicle velocity sensor 24 and 
steering angle sensor 25 acquired in step S125, and 
compares position data obtained by coordinate conver- 
sion of the position data of the obstacle detected by the 15 
obstacle sensor 22 in the previous control period onto 
the coordinate system of the estimated traveling route, 
and the position data of that obstacle acquired in step 
S 125 in the current control period. 

[0136] As a result of comparison, if the two position 20 
data on that coordinate system indicate an identical 
location, it can be determined that the obstacle stands 
still; if they indicate different locations, it can be deter- 
mined that the obstacle is moving. Furthermore, if the 
obstacle is moving, the CPU 101 checks if the direction 25 
of a velocity vector represented by these two position 
data is approaching the traveling route computed in this 
step, thus determining if the obstacle and vehicle are 
moving in directions in which they relatively approach 
each other. 30 
[0137] Note that the estimation method of the 
traveling route is known to those who are skilled in the 
art by, e.g., Japanese Laid-Open Patent No. 10-100820 
previously proposed by the present applicant, and a 
detailed description thereof will be omitted in this 35 
embodiment. 

[0138] Steps S130 to S132: The display threshold 
value Th set in step S126 is compared with the distance 
D to the obstacle detected by the obstacle sensor 22 in 
step S130, and enlarged-scale display is made in step 40 
S131 or the infrared ray image sensed by the infrared 
light camera 23 is directly displayed in step S132 in 
accordance with the comparison result, as in the 
processing in steps S109 to S1 1 1 (Fig. 17) and the dis- 
play modes shown in Figs. 19 to 22 in the sixth embod- 45 
iment. 

[0139] According to this embodiment mentioned 
above, when the auto mode is selected and the zoom 
ON/OFF switch 36 is ON, since the driver can easily 
recognize an obstacle far away from the vehicle, and so 
enlarged-scale display can be started at an earlier tim- 
ing in rough weather that exerts a heavier mental load 
on the driver on driving than in fine weather in accord- 
ance with the maneuvering environment estimated 
based on the operation state of the wipers and ON/OFF 55 
state of the fog lamps, driver's vision can be adequately 
assisted. 

[0140] When the vehicle is turning, since enlarged- 



scale display is made only when the detected obstacle 
and the vehicle are relatively approaching each other, 
the image of that obstacle can be inhibited from being 
displayed in an enlarged scale when the obstacle is rel- 
atively moving in a direction away from the vehicle, the 
driver can be prevented from being distracted. 

[Eighth Embodiment] 

[0141] The eighth embodiment, which is based on 
the aforementioned display apparatus for a vehicle 
according to the sixth embodiment, will be described 
below. In the following description, an explanation of the 
same arrangement as that in the sixth embodiment will 
be omitted, and a characteristic feature of this embodi- 
ment will be mainly explained. 
[0142] Fig. 24 is a flow chart of a display control 
processing by the display apparatus for a vehicle 
according to the eighth embodiment, and shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
power switch 34 is ON. 

[0143] Steps S141 to S143 in Fig. 24: The CPU 101 
executes the same processing as in steps S101 to S103 
in the sixth embodiment (Fig. 17), and if it is determined 
in step S141 that the auto mode is selected, the flow 
advances to step S144. 

[0144] Step S144: The CPU 101 checks if the head 
lamp switch 33 is ON. If YES in step S144 (switch 33: 
ON), the flow advances to step S145; if NO in step S144 
(switch 33: OFF), the flow advances to step S143 to 
stop display of the infrared ray image. 
[0145] Step S145: Since the auto mode is selected 
and the head lamps are ON, the CPU 101 checks if the 
zoom ON/OFF switch 36 is ON. If YES in step S145 
(switch 36: ON), the flow advances to step S146; if NO 
in step S145, the flow advances to step S150 to start or 
continue direct display of the infrared ray image sensed 
by the infrared light camera 23. 
[0146] Steps S146 and S147: The CPU 101 loads 
data corresponding to the output signal from the obsta- 
cle sensor 22 (step S146), and checks based on the 
loaded data if an obstacle is present (step S147). If YES 
in step S147 (an obstacle is present), the flow advances 
to step S148; if NO in step S147, the flow advances to 
step S150 to start or continue direct display of the infra- 
red ray image sensed by the infrared light camera 23. 
[0147] Step S148: If it is determined based on the 
locations of obstacles and the distances D between 
those obstacles and vehicle contained in the data 
acquired in step S146 that only one obstacle is present, 
the CPU 101 checks if the distance D to that obstacle is 
larger than a predetermined low beam irradiation dis- 
tance (range) pre-stored in the ROM 103. On the other 
hand, if a plurality of obstacles are present, the CPU 
101 checks if the distance D to one of obstacles, which 
is located at a position closest to the vehicle, is larger 
than the predetermined low beam irradiation distance. 
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[0148] If it is determined in this step that the dis- 
tance D to the obstacle is larger than the predetermined 
low beam irradiation distance (YES in step S148), since 
it is determined that the driver can hardly visually recog- 
nize that obstacle, the flow advances to step S 149 to 
display the image of that obstacle in an enlarged scale; 
if the distance D to the obstacle is equal to or smaller 
than the predetermined low beam irradiation distance 
(NO in step S148), since that obstacle is present within 
the low beam irradiation range and it is determined that 
the driver can visually confirm the obstacle irradiated 
with a low beam emitted by the head lamps, the flow 
advances to step S1 50 to start or continue direct display 
of the infrared ray image sensed by the infrared light 
camera 23. 

[0149] Steps S149 and S150: The CPU 101 makes 
enlarged-scale display in step S149 or directly displays 
the infrared ray image sensed by the infrared light cam- 
era 23 in step S150, as in the processing in steps S110 
and S111 (Fig. 17) and the display modes shown in 
Figs. 19 to 22 in the sixth embodiment. 
[0150] According to this embodiment described 
above, when the auto mode is selected and the zoom 
ON/OFF switch 36 and head lamp switch 33 are ON, if 
the obstacle detected based on the output signal from 
the obstacle sensor 22 is present beyond the range (dis- 
tance) irradiated with a low beam emitted by the head 
lamps, the image of that obstacle is displayed in an 
enlarged scale, and the driver can easily recognize the 
obstacle. 

[Ninth Embodiment] 

[0151] Fig. 25 is a block diagram of a display appa- 
ratus for a vehicle according to the ninth embodiment of 
the present invention. 

[0152] Referring to Fig. 25, reference numeral 42 
denotes an obstacle sensor such as a CCD (Charge 
Coupled Device) camera, laser radar, millimeter-wave 
radar, or the like for detecting the distance to an obsta- 
cle ahead of the vehicle by a known method. Reference 
numeral 43 denotes an infrared light (infrared ray) cam- 
era for sensing an image of an environment ahead of 
the vehicle using infrared rays. 
[0153] Reference numeral 44 denotes a head/face 
position detection unit which is provided to, e.g., an 
instrumental panel in the passenger room and is used 
to detect the head/face position of the driver. The unit 44 
comprises an infrared projector 44b for projecting infra- 
red rays toward the head/face position of the driver 
seated on the driver's seat, and a near infrared light 
camera 44a for sensing an image of the head/face of 
the driver irradiated with light projected by the infrared 
projector 44b. 

[0154] Reference numeral 45 denotes a vehicle 
velocity sensor for detecting the vehicle velocity as a 
travel state of the vehicle. Reference numeral 46 
denotes a wiper switch with which the driver can 



start/stop wipers provided to the vehicle, and can set its 
operation state such as an operation speed (period) or 
the like. 

[0155] Reference numeral 47 denotes a visible light 
5 camera such as a CCD camera or the like for sensing 
an image ahead of the vehicle. 
[0156] Reference numeral 48 denotes a display 
such as a liquid crystal display, head-up display, or the 
like for displaying an image sensed by the infrared light 
10 camera 43 (to be referred to as an infrared ray image 
hereinafter). Note that the display 48 is preferably laid 
out at a position (e.g., near the central position of a 
dashboard) in front of the driver's seat of the vehicle 
where the driver can easily see the displayed image 
15 without largely moving the line of sight when he or she 
looks forward. 

[0157] Reference numeral 49 denotes a known 
steering counter torque adjustment unit for adjusting the 
steering force required for the driver to steer the vehicle 

20 by operating a handle (steering wheel). 

[0158] Reference numeral 50 denotes a voice out- 
put unit for giving various pieces of information to the 
driver by means of a voice guidance. Reference 
numeral 51 denotes a mode select switch which allows 

25 the operator to select a given display mode of a sensed 
image on the display 48. The display mode that can be 
selected by this switch includes a manual mode which 
displays an infrared light image on the display 48 while 
the operator holds a manual ON/OFF switch 56 ON, and 

30 an auto mode which displays an infrared light image on 
the display 48 in accordance with a display control 
processing (to be described later), when the auto mode 
is selected by the mode select switch 51. 
[0159] Reference numeral 52 denotes a power 

35 switch which allows the operator (driver) to turn on/off 
power supply to a display controller 41 . 
[0160] The manual ON/OFF switch 56 is used to 
display the infrared ray sensed image on the display 48 
while the operator holds this switch ON when the man- 

40 ual mode is selected by the mode select switch 51 . 
[0161] The display controller 41 controls display of 
the infrared ray image on the display 48 on the basis of 
output signals (data corresponding to the output sig- 
nals) from the respective sensors, and the detected 

45 operation states of the operation switches (to be 
described in detail later). 

[0162] The display control processing of the display 
controller 41 is executed by a CPU 101 in accordance 
with software pre-stored in a ROM 103 or the like while 
so using a RAM 102 as a work area. 

[0163] An example of the display control processing 
done by the display controller 41 in this embodiment will 
be described below. 

[0164] Fig. 26 is a flow chart of the display control 
55 processing by the display apparatus for a vehicle 
according to the ninth embodiment, and shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
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power switch 52 is ON. 

[0165] Step S201 in Fig. 26: The CPU 101 checks 
by detecting the operation state of the mode select 
switch 51 if the auto mode is selected. If NO in step 
S201 (manual mode), the flow advances to step S202; if 5 
YES in step S201 (auto mode), the flow advances to 
step S203. 

[0166] Step S202: Since the manual mode is 
selected, the CPU 101 checks if the manual ON/OFF 
switch 56 is ON. If YES in step S202 (switch 56: ON), 10 
the flow advances to step S213 to start or continue dis- 
play of the infrared ray image; if NO in step S202 (switch 
56: OFF), the flow advances to step S212 to stop dis- 
play of the infrared ray image. 

[0167] Step S203: Since the auto mode is selected, 15 
the CPU 101 loads data corresponding to the output 
signal from the vehicle velocity sensor 45 and signals 
representing the operation states of the wiper switch 46 
and a fog lamp switch 53. 

[0168] Step S204: The CPU 101 sets a display start 20 
threshold value Th, which serves as a reference in 
determining if zoom display is made upon displaying an 
infrared ray image on the display 48, with reference to a 
map (table) pre-stored in the ROM 103. 
[0169] As an example of the characteristics of the 25 
display threshold value Th stored in this map, a thresh- 
old value which increases with increasing vehicle veloc- 
ity acquired in step S203 can be set. Alternatively, a 
threshold value which increases with increasing opera- 
tion speed (i.e., decreasing operation period) repre- 30 
sented by the operation state of the wiper switch 46 
acquired in step S203 can be set. Also, a larger thresh- 
old value can be set compared to the OFF state of the 
fog lamp switch 53 when the operation state of the flog 
lamp switch 53 acquired in step S203 represents the 35 
ON state. 

[0170] Step S205: The CPU 101 obtains the loca- 
tions of obstacles ahead of the vehicle and distances D 
between those obstacles and the vehicle by acquiring 
data corresponding to the output signal from the obsta- 40 
cle sensor 42. 

[0171] Step S206: The CPU 101 sets a delay time T 
from a generation timing of an alert (guidance) by the 
voice output unit 50, which indicates the start of display 
of the infrared ray image until display of the infrared ray 45 
image actually starts, with reference to a map (table) 
pre-stored in the ROM 103. As an example of the char- 
acteristics of the delay time T stored in that map, a time 
which decreases with decreasing distance D acquired 
in step S205 can be set. Or a time which decreases with so 
increasing vehicle velocity acquired in step S203 can be 
set. 

[0172] Step S207: The CPU 101 checks if the data 
corresponding to the output signal from the obstacle 
sensor 42 acquired in step S205 includes information 55 
(location and distance D) that represents the presence 
of an obstacle. If NO in step S207 (absence of an obsta- 
cle), the flow advances to step S208; if YES in step 



S207, the flow advances to step S209. 
[0173] Step S208: Even when no obstacle is 
present ahead of the vehicle, if visibility in the forward 
direction is not good depending on the weather condi- 
tion such as heavy fog or the like, the infrared ray image 
should be displayed to help driving of the driver. In this 
step, the CPU 101 checks if the operation state of the 
fog lamp switch 53 acquired in step S203 is ON or the 
operation state of the wiper switch 46 acquired in step 
S203 indicates a low-speed (or intermittent) operation. 
If YES in step S208 (the fog lamps are ON or the wipers 
are in low-speed operation), since it can be estimated 
that visibility in the forward direction is not good, the flow 
advances to step S213 to start or continue display of the 
infrared ray image; if NO in step S208 (the fog lamps are 
OFF or the wipers are not operated), the flow advances 
to step S212 to stop display of the infrared ray image. 
[0174] Step S209: Since an obstacle is present 
ahead of the vehicle, the CPU 101 checks if the dis- 
tance D to the obstacle acquired in step S205 is smaller 
than the display start threshold value Th set in step 
S204, so as to determine if the infrared ray image is to 
be displayed. If NO in step S209 (D > Th), since it can 
be determined that display of the infrared ray image is 
not required, the flow advances to step S212; if YES in 
step S209 (D < Th), the flow advances to step S210. 
[0175] Step S210: The CPU 101 gives the driver a 
voice guidance indicating the start of display of the infra- 
red ray image prior using the voice output unit 50 prior 
to the start of display of the infrared ray image. 
[0176] Step S21 1 : The CPU 101 checks if the delay 
time T set in step S206 has elapsed from the voice out- 
put timing in step S210. If YES in step S211 (the delay 
time T has elapsed), the flow advances to step S213 to 
start display of the infrared ray image; if NO in step 
S211 (the delay time T is being measured), the flow 
advances to step S21 2 to turn off display of the infrared 
ray image. 

[0177] Step S212: The CPU 101 stops or inhibits 
display of the infrared ray image on the display 48, and 
the flow returns. 

[0178] Step S213: The CPU 101 starts or continues 
display of the infrared ray image on the display 48, and 
the flow returns. 

[0179] According to this embodiment mentioned 
above, even when no obstacle is present ahead of the 
vehicle, if it is estimated based on the operation state of 
the fog lamp switch 53 or wiper switch 46 that visibility in 
the forward direction is poor, the infrared ray image is 
automatically displayed. On the other hand, when an 
obstacle is present, the ON/OFF state of display of the 
infrared ray image is adequately controlled in accord- 
ance with the distance D to that obstacle. Hence, driving 
assistance corresponding to the maneuvering environ- 
ment can be adequately offered without disturbing the 
driver. 

[0180] In this embodiment, when it is determined 
that an infrared ray image must be displayed, the voice 
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output unit 50 outputs a voice guidance indicating that 
display is to be started prior to the beginning of the dis- 
play, so the driver can be prevented from taking fright 
when display of the infrared ray image on the display 48 
starts suddenly. 

[0181] Furthermore, since the display start thresh- 
old value Th and delay time T used as references in 
determining if display of the infrared ray image is started 
use optimal values corresponding to the detected vehi- 
cle velocity or the like, flexible display control can be 
done adequately 

[10th Embodiment] 

[01 82] The 1 0th embodiment, which is based on the 
aforementioned display apparatus for a vehicle accord- 
ing to the ninth embodiment, will be explained below. In 
the following description, an explanation of the same 
arrangement as that in the ninth embodiment will be 
omitted, and a characteristic feature of this embodiment 
will be mainly explained. 

[0183] Fig. 27 is a flow chart of a display control 
processing by the display apparatus for a vehicle 
according to the 10th embodiment, and shows the 
sequence of software executed by the CPU 101 while 
the ignition key switch of the vehicle is ON, and the 
power switch 52 is ON. 

[0184] Referring to Fig. 27, the processing in steps 
S221 and S222 are the same as those in steps S201 
and S202 in the ninth embodiment (Fig. 26). 
[0185] Steps S223 and S224: The CPU 101 obtains 
the locations of obstacles ahead of the vehicle and dis- 
tances D between those obstacles and the vehicle by 
acquiring data corresponding to the output signal from 
the obstacle sensor 42 (step S223), and checks if the 
acquired data contains information (location and dis- 
tance D) representing the presence of an obstacle (step 
S224). If NO in step S224 (presence of an obstacle), the 
flow advances to step S225; if YES in step S224 
(absence of an obstacle), the flow advances to step 
S226. 

[0186] Step S225: The CPU 101 checks visibility in 
the forward direction as in step S208 in the ninth 
embodiment (Fig. 26). If it is determined that the visibil- 
ity is poor, the flow advances to step S232; otherwise, 
the flow advances to step S231 . 
[0187] Step S226: The CPU 101 sets a presence 
region of the obstacle in an image sensed by the visible 
light camera 47 (to be referred to as a visible light image 
hereinafter) by associating the obstacle, the presence 
of which is detected based on the data corresponding to 
the output signal from the obstacle sensor 42 in step 
S224, and an image portion corresponding to the obsta- 
cle in the visible light image, and computes a luminance 
contrast C of the set obstacle presence region. 
[0188] Note that the visible light image is digital 
multi-valued image data output from the visible light 
camera 47 itself or generated by an input interface (not 



shown) in the display controller 41 to which the visible 
light camera 47 is connected. 
[0189] A method of setting the obstacle presence 
region in the visible light image wilt be explained below. 

5 [0190] Fig. 28 shows an example of an image 
sensed by the visible light camera 47, and the coordi- 
nate axis set on that image. In this embodiment as well, 
the top view of the detection range of the obstacle sen- 
sor 42 is the same as Fig. 4 in the first embodiment, and 

10 that detection range includes obstacle A (another lead- 
ing vehicle) and obstacle B (pedestrian), which corre- 
spond to the sensed image output from the visible light 
camera 47 shown in Fig. 28. 

[0191] Also, in this embodiment, the detection 

15 results of obstacles A and B by the obstacle sensor 42 
are obtained as a distance Da between the vehicle and 
obstacle A and an angle 6a (« 0) obstacle A makes with 
the central axis, and a distance Db between the vehicle 
and obstacle B and an angle 0b obstacle B makes with 

20 the central axis. 

[0192] Therefore, when the position relationship 
between the central axis of the detection range of the 
obstacle sensor 42, and the coordinate axis of the 
image sensing surface (field angle) of the visible light 

25 camera 47 is determined in advance in correspondence 
with each other, the distance D to an obstacle output 
from the obstacle sensor 42 is counted as the number of 
lines (the number of pixels) from the lower side of the 
image sensing surface, and the angle 0 is counted as 

30 the number of pixels from the central line in the horizon- 
tal direction of the image sensing surface. As a result, 
that pixel (pixel group) in the image sensing surface, 
which is specified by these counts, can be determined 
to correspond to the entire shape of an obstacle 

35 detected by the obstacle sensor 42. 

[0193] Therefore, if such processing is done, the 
obstacle contained in the sensed image from the visible 
light camera 47 can be associated with the distance D 
to that obstacle, and when a rectangle including the 

40 pixel group that forms the obstacle is also set on the 
coordinate system of the corresponding coordinate 
axis, the obstacle detection region included in the visi- 
ble light image can be set in the display controller 41 . 
[0194] An example of the method of computing the 

45 contrast C of the obstacle detection region set in this 
way will be explained below. 

[0195] A visible light image as digital multi-valued 
image data consisting of M x N (M and N are natural 
numbers) pixels undergoes a so-called edge detection 

so processing using a contrast detection filter (3x3 pixel 
matrix) shown in, e.g., Fig. 29, thus computing the con- 
trast values of pixels that form the visible light image. Of 
the computed contrast values, a sum total of the con- 
trast values of pixels which form an image region 

55 (obstacle presence region) that contains the image of 
the obstacle is computed, thus obtaining the contrast C 
of the obstacle detection region. 
[0196] Step S227: The CPU 101 sets a delay time T 
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as in step S206 in the ninth embodiment (Fig. 26). 
[0197] Step S228: The CPU 101 checks if the con- 
trast c computed in step S226 is larger than a predeter- 
mined display threshold value Tha. If YES in step S228 
(C > Tha), since it is estimated that the contrast 
between the obstacle included in the visible right image 
and its surrounding region is high, and the visibility is 
high enough for the driver to visibly confirm the pres- 
ence of the obstacle when he or she looks ahead of the 
vehicle, the flow advances to step S231 to stop display 
of the infrared ray image. Conversely, if NO in step S228 
(C < Tha), since it is determined that the driver can 
hardly recognize the presence of the obstacle due to 
low contrast when he or she looks ahead of the vehicle, 
i.e., visibility in the forward direction is poor, the flow 
advances to the processing in step S229 and subse- 
quent steps to start or continue display of the infrared 
ray image. 

[0198] Steps S229 to S232: The CPU 101 executes 
the same processing as those in steps S210 to S213 in 
the ninth embodiment (Fig. 26). 
[0199] According to this embodiment mentioned 
above, even when no obstacle is present ahead of the 
vehicle, if it is estimated based on the operation state of 
the fog lamp switch 53 or wiper switch 46 that visibility in 
the forward direction is poor, the infrared ray image is 
automatically displayed. On the other hand, when an 
obstacle is present, the contrast C of the obstacle con- 
tained in the visible light image from the visible light 
camera 47 is computed and visibility in the forward 
direction is estimated as a maneuvering environment on 
the basis of the computation result, and the ON/OFF 
state of display of the infrared ray image is adequately 
controlled in accordance with the computation result. 
Hence, driving assistance corresponding to the maneu- 
vering environment can be adequately offered without 
distracting the driver. 

[0200] In this embodiment as well, when it is deter- 
mined that an infrared ray image must be displayed, the 
voice output unit 50 outputs a voice guidance indicating 
that display is to be started prior to the beginning of the 
display, and the driver can be prevented from taking 
fright when display of the infrared ray image on the dis- 
play 48 starts all of a sudden. 

[11th Embodiment] 

[0201] The 1 1th embodiment, which is based on the 
aforementioned display apparatus for a vehicle accord- 
ing to the ninth embodiment, will be explained below. In 
the following description, an explanation of the same 
arrangement as that in the ninth embodiment will be 
omitted, and a characteristic feature of this embodiment 
will be mainly explained. 

[0202] Fig. 30 is a flow chart of a display control 
processing by the display apparatus for a vehicle 
according to the 11th embodiment, and shows the 
sequence of software executed by the CPU 101 while 



the ignition key switch of the vehicle is ON, and the 
power switch 52 is ON. 

[0203] Step S241 in Fig. 30: The CPU 101 checks 
based on the output signal from the head/face position 

5 detection unit 44 if the direction (deviation) of the 
head/face of the driver is equal to or larger than a pre- 
determined value, thus determining if the driver looks 
ahead of the vehicle or aside (inattentively). The method 
of detecting the head/face position of the driver using 

10 the output signal from the head/face position detection 
unit 44 which comprises the near infrared light camera 
44a and infrared projector 44b is a state-of-the-art tech- 
nique, and a detailed description thereof will be omitted. 
The detection method will be briefly explained below. 

15 [0204] Figs. 31 A and 31 B are views for explaining 
the method of detecting the head/face direction of the 
driver by the head/face position detection unit 44 in the 
11th embodiment. 

[0205] Figs. 31 A and 31 B show an image of the 

20 head/face of the driver irradiated with light projected 
from the infrared projector 44b, which is sensed by the 
near infrared light camera 44a. Fig. 31 A shows a case 
wherein the driver gazes forward, and Fig. 31 B shows a 
case wherein the driver looks aside (inattentively). 

25 [0206] As an example of the detection method, 
when an image of the head/face of the driver irradiated 
with light projected from the infrared projector 44b is 
sensed by the near infrared light camera 44a, since the 
pupil portions of the driver are sensed bright due to cor- 

30 nea reflection, the pupil positions (Fig. 31 A) when the 
driver looks forward are stored as the reference posi- 
tions. Then, deviations L of the pupil positions with 
respect to the stored reference positions are detected, 
and if the deviations L have exceeded a predetermined 

35 value, it can be determined that the driver looks aside. 
[0207] As the method of setting the reference posi- 
tions, the sensed image signal output from the near 
infrared light camera 44a undergoes a known binariza- 
tion processing or feature point extraction processing to 

40 extract an image of the head/face portion of the driver, 
and pupil positions which are located nearly symmetri- 
cally in the extracted image of the head/face portion, as 
shown in Fig. 31A, can be detected as the reference 
positions when the driver looks ahead. 

45 [0208] If YES in step S241 (the driver looks aside), 
the flow advances to step S253 to stop display of an 
infrared ray image (an image ahead of the vehicle 
sensed by the infrared light camera 43); if NO in step 
S241 (the driver looks forward), the flow advances to 

50 step S242. 

[0209] Steps S242 and S243: The processing in 
steps S242 and S243 are the same as those in steps 
S201 and S202 in the ninth embodiment (Fig. 26). 
[0210] Step S244: The CPU 101 obtains the loca- 

55 tions of obstacles ahead of the vehicle and distances D 
between those obstacles and the vehicle by acquiring 
data corresponding to the output signal from the obsta- 
cle sensor 42 as in step S205 in the ninth embodiment 
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(Fig. 26). 

[0211] Step S245: The CPU 11 checks if the data 
corresponding to the output signal from the obstacle 
sensor 42 acquired in step S244 includes information 
(location and distance D) that represents the presence 
of an obstacle. If NO in step S245 (absence of an obsta- 
cle), the flow advances to step S246; if YES in step 
S245, the flow advances to step S247, as in step S207 
in the ninth embodiment (Fig. 26). 
[0212] Step S246: The CPU 101 checks visibility in 
the forward direction as in step S208 in the ninth 
embodiment (Fig. 26). If it is determined that the visibil- 
ity in the forward direction is not good, the flow 
advances to step S252 to start or continue display of the 
infrared ray image; otherwise, the flow advances to step 
S253 to stop display of the infrared ray image. 
[0213] Steps S247 and S248: The CPU 101 sets a 
display start threshold value Th and delay time T as in 
steps S204 and S206 in the ninth embodiment (Fig. 26). 
[0214] Steps S249 to S253: The CPU 101 stops 
display of the infrared ray image or outputs a voice guid- 
ance prior to start of display of the infrared ray image in 
accordance with the set display start threshold value 
Th, and then starts display of the infrared ray image 
after an elapse of the delay time T, as in the processing 
in steps S209 to S213 in the ninth embodiment (Fig. 
26). 

[021 5] In steps S254 to S257, the CPU 1 01 controls 
the steering counter torque adjustment unit 49 to gener- 
ate a counter torque for a predetermined period of time 
after the start of display, so as to prevent the driver from 
taking fright (being disturbed) and making unwanted 
steering operation by starting display of the infrared ray 
image on the display 48. More specifically, the assist 
amount of the power steering system may be 
decreased or a rotation torque in a direction opposite to 
the steering direction of the driver may be given. 
[021 6] Steps S254 and S256: More specifically, the 
CPU 101 checks if a predetermined time has elapsed 
after the beginning of display of the infrared ray image 
on the display 48. If YES in step S254 (the predeter- 
mined time has elapsed after the beginning of display), 
the magnitude of a counter torque is set at a default 
value in step S256. 

[0217] Steps S255 and S257: Since it is determined 
in step S254 that the predetermined time has not 
elapsed yet after the beginning of display, the CPU 101 
checks based on the output signal from the steering 
angle sensor 55 if the rotation angular velocity of the 
steering wheel by the driver is equal to or higher than a 
predetermined angular velocity, and the steering angle 
is equal to or larger than a predetermined angle. Note 
that the rotation angular velocity can be obtained by 
computing a change in output signal from the steering 
angle sensor 55 per unit time. 

[0218] If it is determined that the rotation angular 
velocity of the steering wheel is equal to or higher than 
the predetermined angular velocity and the steering 



angle is equal to or larger than the predetermined angle 
(YES in step S255), since the driver is steering the 
steering wheel, and it is not preferable to increase the 
steering counter torque during such operation, the mag- 
5 nitude of the counter torque is set at a default value in 
step S256. 

[0219] On the other hand, if it is determined that the 
rotation angular velocity of the steering wheel is lower 
than the predetermined angular velocity and/or the 

10 steering angle is smaller than the predetermined angle 
(NO in step S255), the steering counter torque adjust- 
ment unit 49 increases the magnitude of the counter 
torque required upon steering to make the driver's 
steering operation harder. 

15 [0220] At this time, the magnitude of the counter 
torque generated by the steering counter torque adjust- 
ment unit 49 preferably increases in proportion to an 
increase in vehicle velocity detected by the vehicle 
velocity sensor 45. For example, a map (table) that pre- 

20 stores such characteristics can be looked up in this step 
in accordance with the vehicle velocity. 
[0221] According to this embodiment, in addition to 
the same effects as in the ninth embodiment, when the 
head/face position detection unit 44 determines that the 

25 driver looks aside, display of the infrared ray image on 
the display 48 is stopped. In this manner, driving assist- 
ance corresponding to the maneuvering environment 
can be adequately offered. 

[0222] When display of an infrared ray image is 
30 started when the driver gazes forward, as the steering 
counter torque adjustment unit 49 generates a counter 
torque for a predetermined period of time to make the 
steering wheel heavy, the driver can be prevented from 
taking fright and abruptly steering the steering wheel 
35 due to start of the infrared ray image on the display 48. 
[0223] The aforementioned 11th embodiment is 
particularly effective when the display 48 displays an 
infrared ray image sensed by the infrared light camera 
43 at a position in front of the driver in a see-through 
40 state. 

[0224] That is, in such see-through display, the 
driver recognizes the infrared ray image sensed by the 
infrared light camera, which overlaps the actual land- 
scape ahead of the vehicle. 

45 [0225] In this case, even while the driver is looking 
aside, if the infrared ray image is kept displayed on the 
display, the driver is distracted. However, according to 
embodiment, when it is determined that the driver looks 
aside, display of the infrared ray image on the display 48 

so is stopped, thus preventing the driver from feeling dis- 
tracted. 

[0226] Furthermore, in this embodiment, when it is 
detected that the driver gazes forward, since a voice 
guidance is output prior to display of the infrared ray 
55 image and the steering counter torque adjustment unit 
49 is controlled to increase the steering force required 
for steering the steering wheel for a predetermined 
period of time after the beginning of display, operability 
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and safety can be further improved when a see-through 
display is used as the display 48. 
[0227] As many apparently widely different embodi- 
ments of the present invention can be made without 
departing from the spirit and scope thereof, it is to be 5 
understood that the invention is not limited to the spe- 
cific embodiments thereof except as defined in the 
appended claims. 

Claims w 

1 . A display apparatus for a vehicle, which is provided 
to a vehicle, and comprises an image sensing 
device (3, 23, 43) for sensing an image of a land- 
scape ahead of the vehicle using infrared rays, a 15 
display (8, 28, 48) for displaying the image sensed 

by said image sensing device at a position in front 
of a driver's seat of the vehicle, and distance detec- 
tion means (2, 22, 42) for detecting an obstacle 
present ahead of the vehicle, and detecting a dis- 20 
tance between the obstacle and the vehicle, char- 
acterized by comprising: 

display control means (1,21, 41) for changing a 
display mode of said display in accordance with 25 
a detection result of said distance detection 
means. 

2. The apparatus according to claim 1, characterized 

in that said display control means (1 ) computes dis- 30 
tances to obstacles contained in the image sensed 
by said image sensing device (3) on the basis of 
that image and the detection result from said dis- 
tance detection means (2), executes a predeter- 
mined image processing of the image so as to 35 
display only the obstacle with the shortest com- 
puted distance to the vehicle, and displays an 
image that has undergone the image processing on 
said display. 

40 

3. The apparatus according to claim 1, characterized 
in that said display control means (1) computes dis- 
tances to obstacles contained in the image sensed 
by said image sensing device (3) on the basis of 
that image and the detection result from said dis- 45 
tance detection means (2), executes a predeter- 
mined image processing of the image so as to 
emphasize and display the obstacle with the short- 
est computed distance to the vehicle compared to 
other obstacles, and displays an image that has so 
undergone the image processing on said display. 

4. The apparatus according to claim 3, characterized 
in that said display control means changes contrast 
between a partial image that represents the obsta- 55 
cle closest to the vehicle, and a surrounding image 
thereof as the predetermined image processing. 



5. The apparatus according to claim 3, characterized 
in that said display control means sets a display 
color of a partial image that represents the obstacle 
closest to the vehicle to be different from a display 
color of a partial image that represents another 
obstacle as the predetermined image processing. 

6. The apparatus according to claim 3, characterized 
in that said display control means sets a luminance 
of a partial image that represents the obstacle clos- 
est to the vehicle to be higher than a luminance of a 
partial image that represents another obstacle as 
the predetermined image processing. 

7. The apparatus according to claim 3, characterized 
in that said display control means emphasizes and 
displays an outline of a partial image that repre- 
sents the obstacle closest to the vehicle as the pre- 
determined image processing. 

8. The apparatus according to claim 1, characterized 
in that said display control means (1) computes dis- 
tances to obstacles contained in the image sensed 
by said image sensing device (3) on the basis of 
that image and the detection result from said dis- 
tance detection means (2), extracts a partial image 
that represents the obstacle with the shortest dis- 
tance to the vehicle, executes a predetermined 
image processing for enlarging the extracted 
image, and displays an image that has undergone 
the image processing on said display. 

9. The apparatus according to claim 1, characterized 
in that said display control means (1) extracts a par- 
tial image that represents a person from an overall 
image sensed by said image sensing device (3), 

computes a distance between the vehicle and 
the person corresponding to the partial image 
on the basis of the overall image and the detec- 
tion result from said distance detection means 
(2), executes a predetermined image process- 
ing of the overall image so as to emphasize and 
display the partial image that represents the 
person with the shortest computed distance to 
the vehicle compared to partial images repre- 
senting other obstacles, and displays an image 
that has undergone the image processing on 
said display (8). 

10. The apparatus according to claim 9, characterized 
in that said display control means determines an 
obstacle, a radiation temperature of which falls 
within a predetermined temperature range, to be a 
person on the basis of radiation temperatures of 
obstacles included in the overall image, prior to 
extraction of the partial image that represents the 
person from the overall image. 
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11. The apparatus according to claim 10, 
characterized by further comprising: 

maneuvering environment detection means (4, 
5) for detecting a maneuvering environment of 5 
the vehicle, and in that 

said display control means corrects the prede- 
termined temperature range in accordance 
with the maneuvering environment detected by 
said maneuvering environment detection 
means. 

12. The apparatus according to claim 11, 
characterized in that said maneuvering environ- 
ment detection means (5) detects as the maneuver- 
ing environment an atmospheric temperature of the 
vehicle, and 

said display control means shifts the predeter- 
mined temperature range toward higher tem- 
perature with increasing atmospheric 
temperature detected by said maneuvering 
environment detection means. 

13. The apparatus according to claim 1 1 , 
characterized in that said maneuvering environ- 
ment detection means (5) detects as the maneuver- 
ing environment calendar information or time 
information, and 

said display control means corrects the prede- 
termined temperature range on the basis of the 
calendar information or time information 
detected by said maneuvering environment 
detection means. 

14. The apparatus according to claim 9, characterized 
in that said display control means changes contrast 
between a partial image that represents the person 
closest to the vehicle, and a surrounding image 
thereof as the predetermined image processing. 

15. The apparatus according to claim 9, characterized 
in that said display control means sets a display 
color of a partial image that represents the person 
closest to the vehicle to be different from a display 
color of a partial image that represents another 
obstacle as the predetermined image processing. 

16. The apparatus according to claim 9, characterized 
in that said display control means sets a luminance 
of a partial image that represents the person clos- 
est to the vehicle to be higher than a luminance of a 
partial image that represents another obstacle as 
the predetermined image processing. 

17. The apparatus according to claim 9, characterized 
in that said display control means emphasizes and 



displays an outline of a partial image that repre- 
sents the person closest to the vehicle as the pre- 
determined image processing. 

18. The apparatus according to claim 9, characterized 
in that said display control means selects a prede- 
termined number of distances which are computed 
between the vehicle and persons corresponding to 
the partial images in ascending order of distance to 
the vehicle, and emphasizes and displays the par- 
tial images that represent the persons correspond- 
ing to the selected distances compared to the 
partial images that represent the other obstacles, 
as the predetermined image processing. 

19. The apparatus according to claim 1, characterized 
in that said display control means (1) extracts par- 
tial images that represent persons from an overall 
image sensed by said image sensing device (3), 

computes distances between the vehicle and 
the persons corresponding to the partial 
images on the basis of the overall image and 
the detection result from said distance detec- 
tion means (2), and 

displays the partial image that represents the 
person with the shortest computed distance to 
the vehicle on said display, and also the com- 
puted distance to the vehicle in association 
with that person near the partial image. 

20. The apparatus according to claim 1, characterized 
in that said display control means (1) computes dis- 
tances between the vehicle and obstacles included 
in the image sensed by said image sensing device 
(3) on the basis of that image and the detection 
result from said distance detection means (2), dis- 
plays a partial image which represents the obstacle 
with the shortest computed distance to the vehicle 
on said display (8), and also displays the distance 
to the vehicle computed in association with that 
obstacle near the partial image. 

21. The apparatus according to claim 1, characterized 
in that said display control means (1) computes dis- 
tances between the vehicle and obstacles included 
in the overall image sensed by said image sensing 
device (3) on the basis of that overall image and the 
detection result from said distance detection means 
(2). 

extracts a partial image that represents a corre- 
sponding obstacle when each of the computed 
distances is larger than a first predetermined 
distance, executes a predetermined image 
processing for enlarging the extracted partial 
image, and displays an image that has under- 
gone the image processing on said display (8). 
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22. The apparatus according to claim 21, 
characterized by further comprising: 

vehicle velocity detection means (24) for 
detecting a vehicle velocity of the vehicle, and 
in that 

said display control means (21) makes 
enlarged-scale display of the partial image eas- 
ier with increasing vehicle velocity of the vehi- 
cle upon executing the predetermined image 
processing. 

23. The apparatus according to claim 22, 
characterized in that said display control means 
makes enlarged-scale display of the partial image 
easier by increasing the first predetermined dis- 
tance with increasing vehicle velocity of the vehicle. 

24. The apparatus according to claim 21 , 
characterized by further comprising: 

turn state detection means (25) for detecting a 
turn state of the vehicle, and in that 
said display control means (21) inhibits execu- 
tion of enlarged-scale display of the partial 
image when said turn state detection means 
detects that the vehicle is turning. 

25. The apparatus according to claim 21, 
characterized by further comprising: 

turn state detection means (25) for detecting a 
turn state of the vehicle, and in that 
when said turn state detection means detects 
that the vehicle is turning and it is determined 
that an obstacle located at a position closer 
than the first predetermined distance is 
approaching along an estimated route of the 
vehicle, said display control means (21) makes 
enlarged-scale display of a partial image which 
represents that obstacle, and 
when said turn state detection means detects 
that the vehicle is not turning, said display con- 
trol means inhibits execution of enlarged-scale 
display of the partial image. 

26. The apparatus according to claim 21, 
characterized in that the first predetermined dis- 
tance is set at a possible irradiation distance of 
head lamps of the vehicle in said display control 
means. 

27. The apparatus according to claim 21, 
characterized in that said display control means 
also displays a symbol mark which informs a pas- 
senger of enlarged-scale display when the partial 
image is displayed in an enlarged scale. 



28. The apparatus according to claim 21 , 
characterized by further comprising: 

maneuvering environment detection means 
5 (31 - 33) for detecting a maneuvering environ- 

ment of the vehicle, and in that 
said display control means (21) corrects the 
first predetermined distance in accordance 
with the maneuvering environment detected by 
10 said maneuvering environment detection 

means. 

29. The apparatus according to claim 28, 
characterized in that said maneuvering environ- 

15 ment detection means (31 , 32) detects a value that 
pertains to visibility of a driver with respect to a 
maneuvering environment ahead of the vehicle as 
the maneuvering environment, and 

20 said display control means corrects the first 

predetermined distance to increase as the 
detection result of said maneuvering environ- 
ment detection means indicates worse visibility. 

25 30. The apparatus according to claim 29, 

characterized in that said maneuvering environ- 
ment detection means (31) detects a wiper opera- 
tion speed of the vehicle as the maneuvering 
environment, and 

30 

said display control means corrects the first 
predetermined distance to increase with 
increasing wiper operation speed detected by 
said maneuvering environment detection 
35 means. 

31. The apparatus according to claim 29, 
characterized in that said maneuvering environ- 
ment detection means (31) detects an ON/OFF 

40 state of fog lamps of the vehicle as the maneuver- 
ing environment, and 

said display control means corrects the first 
predetermined distance to increase when the 
45 fog lamps are ON than a value when the fog 

lamps are OFF. 

32. The apparatus according to claim 21 , 
characterized by further comprising a manual 

so switch (37) which can inhibit execution of enlarged- 
scale display by said display control means (21). 

33. The apparatus according to claim 21 , 
characterized in that said display control means 

55 (21) superimposes an enlarged-scale image of the 
partial image on the overall image as the predeter- 
mined image processing. 
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34. The apparatus according to claim 1 , characterized 
in that said display control means (21) computes 
distances between the vehicle and obstacles 
included in an overall image sensed by said image 
sensing device (23) on the basis of that overall 
image and the detection result from said distance 
detection means (22), 

displays an image obtained by deforming a par- 
tial image that represents a given obstacle on 
said display (28) when the computed distance 
to that obstacle is larger than a first predeter- 
mined distance, so as to make a driver easily 
recognize the presence of the corresponding 
obstacle, and 

displays the overall image on said display when 
the computed distance is smaller than the first 
predetermined distance. 

35. The apparatus according to claim 1, characterized 
in that said display control means (41) computes 
distances between the vehicle and obstacles 
included in an overall image sensed by said image 
sensing device (43) on the basis of that overall 
image and the detection result from said distance 
detection means (42), 



pertains to visibility of a driver with respect to a 
maneuvering environment ahead of the vehicle as 
the maneuvering environment, and 

5 said display control means corrects the second 

predetermined distance to increase as the 
detection result of said maneuvering environ- 
ment detection means indicates worse visibility. 

w 39. The apparatus according to claim 38, 

characterized in that said maneuvering environ- 
ment detection means (46) detects a wiper opera- 
tion speed of the vehicle as the maneuvering 
environment, and 

15 

said display control means corrects the second 
predetermined distance to increase with 
increasing wiper operation speed detected by 
said maneuvering environment detection 
20 means. 

40. The apparatus according to claim 38, 

characterized in that said maneuvering environ- 
ment detection means (53) detects an ON/OFF 
25 state of fog lamps of the vehicle as the maneuver- 
ing environment, and 



displays the overall image on said display (48) 
when the computed distance is smaller than a 
second predetermined distance, and 
stops display of the overall image on said dis- 
play when the computed distance is larger than 
the second predetermined distance. 

36. The apparatus according to claim 35, 
characterized by further comprising: 

vehicle velocity detection means (45) for 
detecting a vehicle velocity of the vehicle, and 
in that 

said display control means changes the sec- 
ond predetermined distance to increase with 
increasing vehicle velocity of the vehicle. 

37. The apparatus according to claim 35, 
characterized by further comprising: 

maneuvering environment detection means 
(46, 53) for detecting a maneuvering environ- 
ment of the vehicle, and in that 
said display control means corrects the second 
predetermined distance in accordance with the 
maneuvering environment detected by said 
maneuvering environment detection means. 

38. The apparatus according to claim 37, 
characterized in that said maneuvering environ- 
ment detection means (46, 53) detects a value that 



said display control means corrects the second 
predetermined distance to increase when the 
30 fog lamps are ON than a value when the fog 

lamps are OFF. 

41. The apparatus according to claim 35, 
characterized by further comprising alert means 

35 (50) for informing a passenger of start of display by 
means of a voice message prior to display of the 
overall image. 

42. The apparatus according to claim 41 , 

40 characterized in that said display control means 
corrects a delay time from an alert generation tim- 
ing by said alert means until the start of display of 
the overall image on said display to be a time which 
decreases with decreasing distance computed for 

45 the obstacle. 

43. The apparatus according to claim 41, 
characterized by further comprising: 

so vehicle velocity detection means (45) for 

detecting a vehicle velocity of the vehicle, and 
in that 

said display control means corrects a delay 
time from an alert generation timing by said 
55 alert means until the start of display of the over- 

all image on said display to be a time which 
decreases with increasing vehicle velocity of 
the vehicle. 
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